Method and system for flexible modeling of a multi-level organization for purposes of assessment

ABSTRACT

A system of hardware and software provides a systematic and systemic method for performing assessment of the effectiveness of different levels within an institution. The system includes template driven definitions that span multiple tiers of hierarchically arranged operating units within the institution thereby providing flexibility and customization to handle a user-definable model of the institution&#39;s organization. In particular, modeling of different nodes of the institution is performed in a flexible manner that may apply to various institutions regardless of their particular organizational structure.

RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication Ser. No. 60/670,963, filed Apr. 12, 2005, entitled“Transaction, Commerce, Community, Content, Learning, and EvaluationSystems”, the disclosure of which is incorporated herein by reference inits entirety. The present application is a continuation of U.S. patentapplication Ser. No. 11/363,868, filed Feb. 27, 2006, entitled “Methodand System for Assessment Within a Multi-Level Organization”, thedisclosure of which is incorporated herein in its entirety.

BACKGROUND

1. Field

The present disclosure relates generally to computer software andhardware systems, and more particularly, to such a system for providingassessment of multiple levels of effectiveness.

2. Background

As part of providing quality educational opportunities, many academicinstitutions are interested in assessing and evaluating theirperformance and efforts as well as the performance of units andindividuals within the institution. These institutions can include, forexample, on-line institutions, K-12 schools, colleges, universities andeven multi-campus university systems. One purpose of assessment involvescollecting and documenting the necessary evidence needed for satisfyingthe requirements of regional or national accreditation bodies. Anotherpurpose is to allow the institution to make informed and objectivedecisions about, for example, curriculum content, course offerings, andeducational effectiveness.

In the past, assessment efforts have been disjointed and performed inrelative isolation at the different levels of the academic institution.For example, the assessment process at the classroom level was performedseparate from any assessment (if ever performed) at a department levelor higher (e.g., institutional level). This approach lead to redundancyof information and efforts, as well as disparate results that would needto be combined in some manner and evaluated if an overall assessmentresult is desired.

Furthermore, identifying and collecting information needed to meetaccreditation requirements may be performed on an ad-hoc basis whenneeded. These processes are then repeated the next time accreditation isreviewed. Such an approach relies heavily on repeatedly reviewing thesame information and generating the necessary documentation each timeaccreditation is reviewed.

Accordingly, there exists a need for an assessment system forinstitutions that is able to consider and connect multipleorganizational tiers of the institution and to provide systematic andsystemic data collection, organization, and retrieval components tosupport institutional assessment efforts.

SUMMARY

Accordingly, one aspect of the present invention relates to a method ofmodeling a plurality of organizational levels in support of conductingan assessment within a multi-level institution. In accordance with thismethod first input from a user is received selecting a node associatedwith one of the plurality of organizational levels and the user ispresented with an editable template for the node. The node may bemodified by completing at least a portion of the editable templatebased, at least in part, on second input received from the user.

Another aspect of the present invention relates to a system for modelinga plurality of organizational levels in support of conducting anassessment within a multi-level institution. This system includes acomputer-based storage including a plurality of editable templates and aprogrammable computer having access to the computer-based storage. Theprogrammable computer is configured to a) receive first input from auser selecting a node associated with one of the plurality oforganizational levels; b) present the user with one of the plurality ofeditable templates for the node; and c) modify the node by completing atleast a portion of the editable template based, at least in part, onsecond input received from the user.

Yet another aspect of the present invention relates to a method forcreating customizable templates for modeling organizational levelsrelated to an assessment within a multi-level institution. In accordancewith this method first input is received from a user identifying one ofthe organizational levels and second input is received from the useridentifying one or more attributes associated with each of a pluralityof nodes associated with the one organizational level. Then an editabletemplate is created, based on the second input, to be used to model oneof the plurality of nodes in the assessment.

One additional aspect of the present invention relates to a system forcreating customizable templates for modeling organizational levelsrelated to an assessment within a multi-level institution. This systemincludes a programmable computer configured to a) receive first inputfrom a user identifying one of the organizational levels; b) receivesecond input from the user identifying one or more attributes associatedwith each of a plurality of nodes associated with the one organizationallevel; and c) create an editable template, based on the second input, tobe used to model one of the plurality of nodes when performing anassessment project.

One more aspect of the present invention relates to a system formodeling a plurality of organizational levels in support of conductingan assessment within a multi-level institution. This system includes acomputer-based storage including a plurality of editable templates; anda programmable computer, having access to the computer-based storage.The computer is configured to a) receive first input from a userselecting a first node associated with one of the plurality oforganizational levels; b) present the user with one of the plurality ofeditable templates for a second node; and c) create the second node bycompleting at least a portion of the editable template based, at leastin part, on second input received from the user.

It is understood that other embodiments of the present invention willbecome readily apparent to those skilled in the art from the followingdetailed description, wherein it is shown and described only variousembodiments of the invention by way of illustration. As will berealized, the invention is capable of other and different embodimentsand its several details are capable of modification in various otherrespects, all without departing from the spirit and scope of the presentinvention. Accordingly, the drawings and detailed description are to beregarded as illustrative in nature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of an institutional assessment system are illustrated byway of example, and not by way of limitation, in the accompanyingdrawings, wherein:

FIG. 1 illustrates a block-level diagram of an institutional environmentin which an assessment system is implemented in accordance with theprinciples of the present invention.

FIG. 2A depicts a flowchart of an exemplary method for using anassessment plan for a multi-tier organization using a template drivenmodel.

FIG. 2B illustrates an exemplary template having hierarchically arrangedforms.

FIG. 2C depicts an exemplary assessment process.

FIG. 2D depicts an exemplary model for a higher education institutionwithin which assessments can be performed in accordance with theprinciples of the present invention.

FIG. 3A depicts a screen shot of a home page from which a user may electto perform functions within an assessment system.

FIGS. 3B-3I depict a series of screen shots of a user interface thatallows a user to perform Goal and Planning Assessment within anassessment system in accordance with the principles of the presentinvention.

FIG. 4A-4C depict a series of screen shots of a user interface thatallows a user to perform curriculum planning in accordance with theprinciples of the present invention.

FIG. 4D illustrates an exemplary curriculum map indicating the alignmentbetween courses and objectives.

FIG. 5A depicts a flowchart of an exemplary method for selectivedeployment of instruments in an assessment management system inaccordance with the principles of the present invention.

FIGS. 5B-5J depict a series of screen shots of a user interface thatallows a user to manage and create instruments within an assessmentsystem in accordance with the principles of the present invention.

FIG. 6A depicts a screen shot of a user interface that relates todisplaying user outcomes in accordance with the principles of thepresent invention.

FIG. 6B depicts a flowchart of an exemplary method for generating thecontents of the screen shot of FIG. 6A.

FIGS. 7A-7K depict a series of screen shots of a user interface thatallows selective reporting of data and content from within an assessmentsystem.

FIGS. 8A-8L depict a series of screen shots of a user interface of anassessment system that provides a number of auxiliary tools.

FIGS. 9A-9D depict a series of screen shots of a user interface of anon-line learning/academic system with which an assessment system can beintegrated in accordance with the principles of the present invention.

The figures include a number of different screen shots of an exemplaryuser interface. One of ordinary skill will appreciate that the depictedinterface is provided by way of example and that the layout, options anditems on each screen shot may be altered or augmented without departingfrom the scope of the present invention.

DETAILED DESCRIPTION

The detailed description set forth below in connection with the appendeddrawings is intended as a description of various embodiments of theinvention and is not intended to represent the only embodiments in whichthe invention may be practiced. The detailed description includesspecific details for the purpose of providing a thorough understandingof the invention. However, it will be apparent to those skilled in theart that the invention may be practiced without these specific details.In some instances, well known structures and components are shown inblock diagram form in order to avoid obscuring the concepts of theinvention. Additionally, the term “automatic” may be used herein todescribe one or more process steps that may be performed in an automatedmanner using various types of programmable processors or computers.However, one of ordinary skill will recognize that the performance ofthese steps may also be accomplished manually or via a combination ofmanual and automated processes.

Terminology

In general, as used herein, “assessment” means a repeating or cyclicimprovement process that involves measuring outcomes versus statedobjectives or goals and identifying corrective action. The term“evaluation” has more of a connotation of a point in time measurement ofone or more outcomes versus stated objectives without necessarily havinga built-in improvement step. For example, a program may have the goal ofdeveloping students' critical thinking abilities and might annuallydetermine whether, as a program, that goal has been achieved. Therefore,one might wish to annually assess this by looking at the aggregateperformance of the program's students on a senior capstone project.

Other terms that are used consistently herein include:

Mission: A broad statement that describes the over-arching purpose of anorganization. Mission statements typically are not measurable because ofthe scope that they encompass and because they are not time-constrained.

Goal: Missions are frequently broken down further into a series ofgoals. Though more specific than a mission, goals are still broadstatements and may not be easily measurable. Goals provide guidance onareas that should be addressed through specific, measurable objectives.

Objective: An objective is an expression of the intended result orconsequence of some activity (e.g. instruction or some otherperformance). It differs from an outcome, which is the achieved result,although the two are sometimes used interchangeably. The activitiesdescribed by an objective should be measurable within a definedtimeframe. Frequently, the term is used with a modifier to clarify theactivity, e.g. a learning objective or a performance objective.

Standard: In educational assessment, for example, a standard is astatement of what a learner should know or be able to do. A standard ismore specific than a goal and should be measurable. In that sense, astandard is much closer to an objective than a goal. However, unlike anobjective, a standard may not be related to a specific activity,instruction, performance, etc. or timeframe. Also, standards arefrequently generated by organizations external to the academicinstitution (e.g. state governments, accrediting agencies) as a means tostandardize measurement of performance or learning across multipleorganizations.

Outcome: An outcome is the achieved result or consequence of someactivity (e.g. instruction or some other performance). It differs froman objective, which is the intended result, although the two aresometimes used interchangeably. Frequently, the term is used with amodifier to clarify the activity, e.g. a learning outcome or aperformance outcome.

A “template” is a means of defining the structure of various data in thesystem and the instruments used to collect that data. A template maydefine one or more customizable forms and the attributes that arepresent on each form. A “form” is a customizable page within a templateconsisting of one or more attributes. An “attribute” is a piece of datathat defines an entity (such as the name of a user) which may berepresented on a form as a field. Certain attribute fields may berequired by the system. The level of customization (e.g. the ability tochange labels on an attribute versus the ability to add new attributesto a form) may be dependent on the type of template that the form is apart of.

For example, a template for collecting data about a user might consistof one form. On that form are specific attributes, such as “name”“social security number” and “type.” The “name” and “social securitynumber” attributes might allow for entry of text information, while the“type” attribute might offer a selection of ‘faculty,’ ‘staff,’ or‘student.’ The “name” and “social security number” attributes might berequired, meaning that any person completing the form must enter datafor those attributes, while the “type” attribute might be optional. Oneof ordinary skill will recognize that there are numerous different andvarying attributes that can be defined for each form using standardavailable web-based interfaces. For example, the information related toan operating unit will vary for a “College” operating unit and a“Department” operating unit. A College operating unit may have fields todesignate Dean, while a Department would have fields to identify thecurrent Chair of the department. Beyond such labeling distinctions,different operating units may even have different types of data. Forexample, an operating unit representing a Center for Media Students mayneed to store, as part of the operating unit, information about thegrant that funds the Center, the grant's initiation and expirationdates, and who manages the Grant, while a Department, funded out of theuniversity budget, may only need to store university budget codesassigned to the Department. Based on their purpose and level within theorganization, different operating units will have differentinformational needs.

Templates may also consist of more than one form; this is particularlyuseful for the representation of a data collection process that consistsof multiple steps. For example, an Assessment Project template mightconsist of a “properties form” to collect data about the properties ofthe project itself, one or more “objective forms” to collect data abouteach objective being measured, and four “activity forms” per objectiveto collect data about the different steps of assessing the performanceof the objective. Each form would have specific attributes defined. Thisexemplar template is illustrated in FIG. 2B. One of ordinary skill willrecognize that there could be many different ways to configure amultiple-form template to represent a variety of different processes fordifferent purposes.

While many examples are provided herein that specifically include ahigher-education institution, the principles of the present inventioncontemplate other types of institutions as well. For example,corporations, governmental entities, and K-12 institutions are allconsidered within the scope of the present invention. An institution mayalso be a consortium of schools and/or campuses. In general terms, aninstitution is an operating unit and is, itself, made up of differentoperating units that may correspond to campuses, colleges, departments,sub-departments, etc. The systems and methods described herein do notrequire any particular arrangement of operating units but, instead,allow the institution to model its organization into a hierarchy ofoperating units for purposes of management, planning, and reporting ofassessment efforts. The terms “unit”, “operating unit”, and “operationalunit” are often used interchangeably herein. However, there are uniqueaspects of operating units related to their respective level in theorganizational hierarchy that allow customization and flexibility basedon these unique aspects.

In the higher education example, operating units offer programs ofstudy, or simply “programs”, with different courses and sections. Thus,assessment can occur, for example, at or across an institutional level,a departmental level, a program level, an individual level, a courselevel and at the section level. Programs may also include coursesoffered by different operating units. For example, a particular programof study may require both an introductory Engineering course and acomputer science course. Within the institutional level assessment,different varieties of assessment may also be performed on differentlevels of operating units as well. For example, the institution may beorganized according to operating units that represent department levels,college levels, etc. Thus, a framework is described herein, in which ahierarchically arranged group of organizational units and activities aredefined and assessed against various criteria and objectives.

FIG. 1 depicts a functional block diagram of an exemplary environmentfor an assessment system 102 in accordance with the principles of thepresent invention. As described in more detail herein, the assessmentsystem 102 provides a framework for performing various types ofassessments of an institution such as, for example, a higher educationinstitution. A user 104 of the system 102 typically uses a web browseror similar interface to communicate with an appropriately configuredfront-end 106 of the system 102. For example, the front-end 106 may be aweb server hosting a number of applications 108 that the user 104 mayaccess. The applications 108 are one or more software components orprograms that execute on a programmable computer platform to providefunctionality related to performing institution-related assessmentactivities. Such applications 108 may include components for modelingthe institution or organization, defining assessment projects,identifying users (e.g., students), defining objectives and goals, andcollecting and reporting data.

The applications 108 may also access data storage facilities 112 andother computer systems 114. For example, the data facilities 112 may beone or more databases having assessment project data stored and arrangedin a convenient and appropriate manner for easy manipulation andretrieval. The other computer systems 114 may be a variety ofthird-party systems that contain data or resources that are useful forthe assessment system 102. In the exemplary higher educationenvironment, the systems 114 may include a student information system(SIS) that maintains student demographic information as would beappreciated by one of ordinary skill. The systems 114 may also includean electronically maintained class, or course, schedule for theinstitution that includes information about the courses such as sectionnumbers, professors, class size, department, college, the studentsenrolled, etc. Other campus-related systems such as financial aid andthe bursar's office may be included in the systems 114 of FIG. 1. Theback-end 110 is appropriately configured software and hardware thatinterface between the applications 108 and the various resources 112,114.

Another resource to which the back end 110 may provide connectivity is acampus (or institutional) academic system 116. An example of such asystem is provided by the present Assignee under the name AcademicSuite™ with many features thereof described in U.S. Pat. No. 6,998,138entitled “Internet-Based Education Support System and Methods”, thedisclosure of which is incorporated herein in its entirety. As describedin that patent, the campus academic system 116, in an academicenvironment, provides a platform that allows students and teachers tointeract in a virtual environment based on the courses for which thestudent is enrolled. This system may be logically separated intodifferent components such as a learning system, a content system, acommunity system, and a transaction system. An example of such a studentis the user 118 that can access the academic system 116 via a webbrowser or similar interface. The user 118 may also be faculty, staff oran administrative officer. An example of these separate components ofthe system 116 are described in detail in pending patent applications“Method and System for Conducting Online Transactions (Ser. No.10,373,924 filed Feb. 25, 2003), “Content and Portal Systems andAssociated Methods” (Ser. No. 11/142,965 filed Jun. 2, 2005), and“Content System and Associated Methods” (Ser. No. 10/918,016 filed Aug.13, 2004), all of which are incorporated herein by reference in theirentirety.

Of particular usefulness to the assessment system 102, the academicsystem 116 provides a virtual space that the user 118 may visit toreceive information and to provide information. One exemplaryarrangement provides the user 118 with a home page where generalinformation may be located and that has links to access course-specificpages where course-specific information is located. As explained in theincorporated patent and patent applications, electronic messaging,electronic drop boxes, and executable modules may be provided within theuser's virtual space on the academic system 116. Thus, with respect tothe assessment system 102, one of the applications 108 may be used togenerate information that is to be deployed to one or more users of theacademic system 116. Via the back-end 110, the information may be sentto the academic system 116 where it is made available to the user 118just as any other information is made available. Similarly, from withinthe academic system 116, the user may enter and submit data that isrouted through the back end 110 to one of the applications 108. One ofordinary skill will recognize that the academic system 116 and theassessment system 102 may be more closely integrated so that theconnectivity between the applications 108 and the system 116 is achievedwithout a network connection or special back end software 110.

Although the front end 106, applications 108, and back end 110 of theassessment system 102 are each depicted as a single block in FIG. 1, oneof ordinary skill will appreciate that each may also be implementedusing a number of discrete, interconnected components. As for thecommunication pathways between the various blocks of FIG. 1, a varietyof functionally equivalent arrangements may be utilized. For example,some pathways may be via the Internet or other wide-area network, whileother pathways may be via a local-area network or even a wirelessinterface. Also, although only a single user 104 of the assessmentsystem 102 is explicitly shown, multiple users are not only contemplatedbut are very likely within the environment of FIG. 1. The structure ofFIG. 1 is logical in nature and does not necessarily reflect thephysical structure of such a system. For example, the assessment system102 may be distributed across multiple computer platforms as can thedata storage 108. Furthermore, the three components 106, 108, 110 areseparate in the figure to simplify explanation of their respectiveoperation. However, these functions may be performed by a number ofdifferent, individual components, or a more monolithically arrangedcomponent. Additionally, any of the three logical components 106, 108,110 may directly communicate with the academic system 116 without anintermediary. Also, although the users 104, 118 are depicted as separateentities in FIG. 1, they may, in fact, be the same user or a single webbrowser instance concurrently accessing both the assessment system 102and the academic system 116.

Assessment within an institution such as a higher-education academicinstitution is a complex undertaking that encompasses many differentlevels of evaluation, data collection, and correction. For example, atthe institutional level, a university may be focused on assessingaccreditation requirements and strategic planning initiatives. At theprogram level, the relevant managers may be focused on assessing programeffectiveness and discipline-specific accreditation requirements. At theclassroom level, the instructors may be interested in assessing coursedesign and curriculum coverage. Thus, some of the diverse questions thatone or more assessment processes may try to address are decisions onfaculty promotion and tenure, determining an effective mix of full-timeand adjunct faculty, determining an effective mix of traditional andremote courses, and assessing whether the needs of specific studentpopulations (e.g., commuters, transferees, minorities) are being met.

Assessment Projects and Initiatives

In general the phrase “assessment project” is used herein to describeefforts of an institution, an operating unit, a program or a course toassess and evaluate different aspects of its operation andeffectiveness. One or more assessment projects may be ongoing at anytime for the same operating unit or different operating units. An“assessment initiative” may encompass a number of different assessmentprojects that are related to a common theme. For example, an assessmentinitiative for Accreditation Board of Engineering and Technology (ABET)accreditation may be defined by the College of Engineering such that theABET assessment project of each of the individual departments within theCollege (e.g. Department of Electrical Engineering, Department of CivilEngineering, Department of Mechanical Engineering, etc.) may connect tothe College-level assessment initiative to enable College-levelreporting on status and outcomes. By explicitly connecting, orassociating, a number of assessment projects with an assessmentinitiative, a user of the present system is able to aggregate andorganize data from among the different projects so that assessment andanalysis can occur from both an individual project perspective as wellas from the assessment initiative's cross-project perspective. Exemplaryassessment projects might be:

a) internal to the program, such as a curriculum planning project or aprogram review;

b) internal to the institution, such as fulfilling institutionalstrategic planning requirements; or

c) for external review or accreditation, such as completing aninstitutional accreditation report or a discipline-specificaccreditation report.

Assessment projects may be initiated at different times and will spandifferent time frames. For example, while a project supporting theunit's responsibilities for institutional accreditation may actuallytake place over several years, strategic planning assessment projectsmay only last eight months and happen with more regularity thanaccreditation.

Even though the assessment process can be highly complex, there are somegeneral provisions of the assessment system 102 of FIG. 1 that allow auser 104 to systematically and automatically initiate assessmentprojects. The general method of FIG. 2A is utilized, regardless ofwhether the user is defining an assessment project for an institutionoperating unit, some lower-level operating unit, or a program or course.

From a starting page, the user makes a selection, in step 202, via auser interface to enter a goal and planning component of the assessmentsystem 102. After activating such a component, the user is prompted todefine an assessment project. To simplify this process, the system 102may provide the user with a default template or form that has somepredefined fields and information. Alternatively, the user may bepresented with a number of different types of templates and asked tomake a selection. For example, the user may be asked whether theassessment project involves a program, a department, a college, aninstitution, or a course. Based on the user's response, an appropriatetemplate may be provided to the user. For example, at the course level,the user would likely not be interested in defining strategic planninggoals. Similarly, at the institutional level, the user would likely notbe interested in specifying a course objective. Thus, the templatesprovided to the user may have their initial content dependent on whattype of assessment project is being defined. In one particularembodiment, the assessment system 102 may be initially provided with anumber of predefined templates. As part of the initialization process(before assessment projects are defined) a user or other administratormay customize these templates in accordance with the institution and itsprograms, departments, courses, etc. The customization of templates isnot a necessary step before an assessment project is begun but itadvantageously provides flexible customization geared towards eachrespective institution and their hierarchical organization.

As a result, in step 204, the user selects a template for the type ofassessment project being generated. Then, in step 206, the user collectsand enters data required by the forms as defined in the selectedtemplate. For example, the collection and entering of data may includemetadata about the assessment project, the specific objectives of theassessment project, and activities related to each objective. Once theform is completed, then it can be presented to other users for reviewand approval in step 208. For example, a department chair may haveresponsibility for creating an assessment project but there may be acommittee that may modify and approve the assessment project before itcan be implemented.

In the above description, it is assumed that the user has adequateaccess rights within the assessment system 102 to perform the desiredtasks. As known to one of ordinary skill, user identities and user rolesmay be employed to restrict user activities so that certain users havemore access privileges to data resources than other users. These accessprivileges may be relatively static or may be more dynamic in nature asvarious users are placed in different roles or committees. Certain usersmay be allowed no access to an assessment project, read-only access tothe project, edit-access to the assessment project, or create-access tothe assessment project. Additionally, within an assessment project, theindividual components of the project may have controlled access based onuser roles as well. Furthermore, an assessment project manager role maybe employed, and assigned to a user, to help with administering andmanaging the project and its implementation. Thus, different resourceswithin the present assessment system may be protected through theapplication of access privileges that allow each resource to beavailable to one or more identified users or shared by all users.

FIG. 2B depicts an exemplary template structure that may be used toimplement the default templates that are presented to the user. Inassessment projects, templates are used to define the set of forms thatare employed to collect the information that makes up the assessmentproject. An assessment project template 250 may include one or more ofseveral types of forms:

a) a main form 252 is used to collect information about the projectitself. Additional fields can be added to a project form in newtemplates by the user, but certain fields may be required on anassessment project template's project form in order to standardize datacollection. While not absolutely necessary, it is advantageous to permitonly one project form per assessment project template 250.

b) An objective form 254 is used to collect information about theobjectives of the assessment project. An assessment project typicallyhas one or more objectives, so at least one objective form is present.Many assessment projects have multiple objectives, however.

c) Objective forms can have one or more activity forms 256-262associated with them in the template 250. These activity forms 256-262collect information about various activities related to conducting theassessment of the objective. Each activity form may have a different setof fields on it. Addition of an objective (e.g. addition of an objectiveform 254) to a template 250, also generates the associated activityforms 256-262.

The template 250 of FIG. 2B may be fully customizable and definable by auser. However, in some instances, an institution may enforcerequirements that all assessment projects include certain baseinformation. Such a requirement may be helpful, for example, in ensuringthat institution-wide reporting can be performed at least on the baseset of assessment project attributes.

In one example, the template 250 of FIG. 2B includes:

Summary Statement: This provides an area to enter a general descriptionor summary of the whole assessment project. This summary can also beused as part of a report generated for the assessment project. Forexample, the summary statement could include an overview of the project,names and titles of the university faculty and staff involved, and asummary of the findings. This might appear as the first page of areport.

Start/End Dates: Projects to evaluate learning outcomes or reviews ofprograms at an institution may have specific time periods which theyspan (e.g. an accreditation project may last several years, but aprogram review may happen annually.) A user initiating an assessmentproject can specify start and end dates for the assessment project. Thedefault start date would be the project's date of creation. The defaultend date would not be set (i.e. no end date) as some assessment projectscould be continuous and ongoing.

Phase Summaries: For each phase that is defined for the assessmentproject by the template, the user (or manager) is able to enter summaryinformation for that phase.

Recurrence: The assessment process is cyclical, so frequently the same(or similar) assessment project may be run on a regular basis. Forexample, a departmental program review may occur every year, or thedepartment's discipline-specific accreditation may need to bere-confirmed every three years. It is therefore useful to track if anassessment project is a re-occurrence of some previous assessmentproject.

Workflow: Workflow status can be used to control the state of thetemplate and to determine what action or activity is next in the processflow.

The flowchart of FIG. 2C provides one example of one way to define aprocess of initiating the assessment process that is also furtherdescribed later with reference to FIGS. 3A-3I. More particularly, theflowchart depicts a specific sequence of events that do not necessarilyhave to occur in that order. For example, the step of modeling (step270) does not necessarily have to occur before an assessment project(step 272). Thus, the steps of the flowchart may occur in an order otherthan depicted in FIG. 2C. There are events that may advantageously beperformed in a particular order for logical reasons. For example,aligning course content (step 280) will usually be performed afterdefining unit objectives (step 274). Additionally, one or more steps ofthe flowchart of FIG. 2C may be added or omitted without departing fromthe scope of the present invention.

Before beginning the definition of an assessment project, theinstitution itself may be modeled. FIG. 2D is a sample of only a part ofa model that may apply to an institution such as, for example, auniversity with multiple campuses. As illustrated, the model capturesdetails about various hierarchically arranged operating units 290-293,as well as programs 296, courses 294, and sections 295. Other equivalentterminology that may also be used herein to describe the organization ofthe institution may specify “nodes” within different “organizationallevels” of the institution. For example, the institution itself may be anode at the organizational level of “institution”. Additionally, at theorganizational level of “campus”, there may be multiple nodes with eachnode representing a different campus (e.g., Campus1 291 of FIG. 2D).Other organizational levels, as already mentioned, may include manydifferent types of operating units and entities within the institutionsuch as, for example, colleges, departments, programs, colleges,sections, courses, etc. all of which can be considered a node. The modelof FIG. 2D is provided merely by way of example to show the flexibilityand customization provided by the present assessment system. Themodeling of the institution may be approached in a variety of ways. Forexample, an academic institution may be comprised of a number ofhierarchically arranged organizational levels. The institution may havea number of different colleges that each have different departments thatoffer various courses having individual sections. Also, within thisorganizational structure, there may be programs of study such as, forexample, different degree programs (B.S., M.S., PhD) or concentrationsof study. One of ordinary skill will recognize that the institution maybe organized in a variety of different ways and that implementations ofthe present invention are able to model those various structures asdifferent operating units that are used within an assessment project orinitiative. Because of the hierarchical nature of the model, operatingunits may have lower-level operating units that are sometimes referredto herein as sub-operating units or lower-level operating units. Theactual mechanics of modeling the institution may be accomplished in avariety of ways without departing from the scope of the presentinvention. For example, as part of the initial set-up of the assessmentsystem described herein, an administrator may manually define thehierarchical arrangement of the institution in a manual step-by-stepprocess. Alternatively, the model definition may take place such that alist of the organizational levels is processed as a batch to create themodel. Still yet another alternative may involve integration with thecampus academic system 116 (or other existing system) through anautomated process to define the model. In one particular example, thecommon attributes for each node within an organizational level may becaptured in the form of a editable template. When an instance of aparticular node is to be modeled within that organizational level, auser can select the appropriate template and then modify it inaccordance with the particular characteristics of that node.

Thus, in step 270, initiation of an assessment project may begin bymodeling the hierarchical structure of the institution. Once this isaccomplished, definition of an assessment project may begin, in step272. The assessment project within an operating unit (e.g., a college, adepartment) may be related to a higher-level operating unit for whichgoals and objectives have already been defined. In an educationalinstitution, for example, the sub-units may be academic departments,administrative or other departments. Underneath these units may becourses or course sections. When creating other assessment projects, theprevious modeling of the institution (step 270) does not need to bere-accomplished but may be re-utilized.

For the sub-operating units, their own objectives are defined in step274. These objectives may be aligned with the top-level goals andobjectives to create a link between the multiple hierarchical levels ofoperating units. Within a sub-operating unit, one or more courses aredefined in step 276. As part of defining a course, the course objectivesare aligned with higher-level goals and objectives, in step 278, so thata link between courses and the meeting of certain goals and objectivescan be established. In addition to the course having its own objectives,the course can also include a definition of its content (e.g.,assignments, tests, reading material, collaboration exercises, etc.). Adescription of the various content for a course may include which courseor unit objective it is aligned with. Thus, in step 280, course contentis aligned with course objectives and, possible, higher-level goals andobjectives. As a result, top-level managers can establish which specificresources satisfy strategic goals and objectives, while low-levelmanagers can identify deficiencies in course offerings and content whencompared to desired curriculum objectives. One advantageous benefit ofthe present system, described below, is that at each level of theprocess an appropriate form can be presented to the user to direct thedefinition of the assessment project.

Goals, Objectives, Standards, and Assessment Planning

FIGS. 3A-3I provide a series of screenshots of one exemplary userinterface that allows a user or project manager to create an assessmentproject. One of ordinary skill will appreciate that not all stepsdepicted in the screen shots are necessary nor are all the featureswithin the user interface.

FIG. 3A depicts a screen shot of an opening window 300 for an Englishdepartment of a university. This window would typically be presented toa user once that user logs into the assessment system and would be basedon the user's identity (e.g., an Engineering professor or departmenthead would be presented with a different, appropriate window). The titlebar 302 shows the example hierarchy of:

University

-   -   College of Liberal Arts        -   English Department            that has been originally defined for this institution or            changed through subsequent modification. Such a hierarchy            can, for example, be defined by the institution when            initially configuring the assessment system or modified            during a later update of that definition. From this “home            page” a user may start a new assessment project or interact            within a current assessment project.

A number of sub-windows may be provided within the opening window 300.Some exemplary windows, as shown in FIG. 3A, include a mission statementwindow 306, a list of available assessment projects 312, and a list ofcourses 308 within the English Department. As shown, the availablecourses 308 can be filtered according to the term in which they areoffered. Another possible sub-window lists the different programs 304offered by the English Department. From the Home window 300, the usercan opt to manage the assessment process by selecting to open the “Goals& Assessment Planning” tab 310. The tabs located in the left of thewindow 300 allow a user to select the different functions availablewithin the system.

The “Goal Manager” window of FIG. 3B is identified by its title bar 302and the tab 314. (This exemplary window is for a hypothetical Business &Finance Department unlike the previous window which related to anEnglish Department). From within this interface, a user is presentedwith a list of analysis processes 318 that are available to supportstrategic planning of a particular unit or program (e.g., institution,campus, department, Graduate program, etc.) These processes can include,for example, “Goals” or “Mission Statement”, “Vision Statement”, andother widely used analysis processes that are known within the area ofstrategic planning. As used herein, a mission statement generallyrelates to what a institution plans to do while a vision statementrelates to the institution's principles or what it believes in.

This list of processes may also include an identification of anappropriate term 316 to which each process relates. For example, thisallows historical mission/vision statements to be maintained even as newones are developed. The current user is also provided with a respectiveicon 317 that is associated with each process that indicates that user'saccess privileges for the particular process.

From the list 318, a user may choose to define or edit a mission/visionstatement for the operating unit or program. As seen in FIG. 3C, theuser is presented in the area labeled “1 Properties” with a mechanismfor specifying whether the statement applies to the entire academicdepartment or some specific context (e.g., program, resource center,etc.) within the department. The text box 323 allows the user to enterthe specific context if appropriate. This box 323 may also be populatedwith predetermined contexts from which the user can select as well. Theterm “department” is used generically and can apply in general towhatever level operating unit the user has selected. In this example,the title bar 302 indicates that the “department” is the “Division ofBusiness & Finance”.

The text box 322 allows the user to enter the language of the missionstatement. In this instance, the vision statement is something differentthan the mission statement and so a separate text box 320 allows theuser to enter the vision statement for the department. Once the missionand vision statements are created, the goals for the department can bedefined using an interface such as the one shown in FIG. 3D.

This interface screen provides a list of the department goals thatincludes a label 324 for each goal and a portion of the text 326 of eachgoal. To add a new goal, an “Add” button 325 is provided. Although notshown, the add process would typically include defining a label for thenew goal, a textual description of the goal, and some additionalmeta-data such as its priority, a category identifier, a keyword (toassist with searching or customized reports), and an identifier ofrelated goal or objectives. These additional meta-data may also beincluded in the list via a user-controlled selection process 327. Inmany instances, entering the mission and vision statements for thedepartment, as well as the goals, may already have been performed andthe user may bypass such steps and move directly to defining a newassessment project.

The screen of FIG. 3E includes a title bar 328 indicating to the userthat the screen pertains to assessment projects of the EnglishDepartment. The list of currently defined assessment projects caninclude a identification of the project 330, its description 332, thename of the initiative to which it relates 334, as well as othermeta-data. However, some of these columns may be eliminated as well. Byselecting a project name 330, the user can open that assessment projectand read or edit the particular data pertaining to that assessmentproject (assuming they have the proper access privileges). Also, usingthe “Add” button 336, a user can begin the process of defining a newassessment project. The defining of a new assessment project may takeplace as described above with the use of templates to assist in theprocess.

FIG. 3F depicts a partially defined assessment project related to“undergraduate Degree Evaluation”. In this window, the user is presentedwith the already defined objectives for the project. For example, a listof objective labels 344 and associated descriptive text 346 are shown.Furthermore, the selection window 347 allows a user to add additionalcolumns to the depicted list. Particular objectives may be added orremoved from the list by use of the “Add” button 340 and the “Remove”button 342.

There are many possible customizable attributes or meta-data that can beassociated with an objective within an assessment project; theseattributes are defined in the various templates for an Objective from(see FIG. 2B). The interface screen of FIG. 3G depicts some exemplaryattributes for an objective. This particular objective is labeled“Non-Print Media” and relates to the study and comparison of non-printmedia. In the “Properties” area 348, a label and a description of theobjective is entered by the user. The description explains the purposeof the objective. The “options” area 350 allows the user to specifyother attributes about the objective. Such as whether it is an academicobjective rather than an administrative objective. This particularinterface screen is related to the template 250 of FIG. 2B in that thescreen, or web page, is presented to a user with a number of forms to becompleted in which each form is defined within the template Thus, theweb page displays the objective form 254 and by selecting differenttabs, the user can select different forms 256-262 to complete or edit.In other words, an Assessment Project template may be thought of as acomplete process (e.g., summary, objective, measurement, findings,follow-up, review) accomplished with a set of interrelated forms.

The alignment of objectives with other goals and objectives is oneadvantageous benefit of the present system. The selection box 352 allowsthe user to select various available objectives and goals that have beendefined. For example, departmental goals, institutional goals, and otheroperating unit goals and objectives can all be selected using the dropdown box 352. Once the particular level of goals is selected, thespecific goals can be selected using the arrow button 354. The selectedgoals and objectives are shown in the box 349. This process allows theuser to identify other goals, objectives, or standards are aligned withthe present objective being defined. Thus, an association can be createdbetween the goals of any of the different hierarchical levels ofoperating units defined within the assessment project and a newlydefined objective. One result is that a user at the institutional level(or some other level) can identify which lower level objectives relateto which of the strategic goals and objectives of the institution. Ingeneral, the result is that associations or alignments may beaccomplished between goals and objectives of higher, lower, and parallelhierarchical levels within the organization or institution.

Within the window 349, an external standard is shown as one of theselected goals. External standards are not necessarily defined by theinstitution but may be defined by a regulatory body or other third partysuch as an accreditation council. In such a case, the drop downselection window 352 would also allow the user to select from amongavailable external standards and have the standard's goals listed forpossible selection. These standards may change over time and, therefore,can be imported into predetermined storage locations so that theinterface screen of FIG. 3G can populate the selection windows with thelatest information as it is provided by the standard's authors.

Other areas 353, 355 of the interface screen allow the user to providecomments or attachments that are related to the objective. Such areas353, 355 may be particularly helpful when the objective is beingreviewed by a committee or other party.

Operating Units

The above-described defining of goals, assessment projects, andobjectives relies on the defining of operating units. The present systemand methods provide techniques for defining operating units in aflexible manner that allows an institution to model whatever itsorganizational hierarchy structure looks like. The interface screen ofFIG. 3H depicts one exemplary method for a user to define operatingunits, programs, and courses within the “Division of Art”. Using such aninterface allows campuses, colleges, departments, and sub-departments todefine their own hierarchy.

The screen of FIG. 3H presents a list of available, or defined, units,programs, courses, and sections 360 that exist for the Division of Art.Optionally, visual cues such as icons 364, 366, 368, may be provided sothat the user can determine whether an item in the list is a course, aprogram or a unit. From the tool bar 362, a user can elect to add a“unit”, a “program” or a “course” at the particular hierarchicalposition (e.g., Division of Art). To move to a different level, the userwould select an appropriate choice from the lists of units that arepresented. Further details of these items are described later.

The identification and assignment of roles to different users isimportant to determining the access privileges to various data resourcespreviously described. The screen of FIG. 3I depicts one exemplary way tomanage users within a department. For example, the “user” button 372allows a user to elect to add a new user to the department. In thisprocess, information about the user would be provided. Such informationcan include, for example, name, email address, user ID, role, accessprivileges, etc. Additional columns can be used as well via auser-selection process such as window 376.

The screen of FIG. 3I shows a list 374 of existing users and theirassociated meta-data. This list 374 can be manipulated so that all userscan be listed or only certain, selected users can be listed. Thus, usingthe drop-down window 370, a user can elect to display within the list374 only those users satisfying a particular role.

Curriculum Planning and Mapping

The previously described interface screens relate to the generalcategory of goals and assessment planning for an operating unit. Suchplanning usually takes place at the relatively higher layers of theinstitution and relate to the strategic goals and objectives of theinstitution. As shown, however, lower-level operating units can defineobjectives and goals as well. Finally, at the lowest hierarchicallevels, the objectives which are defined relate back, or are aligned to,the previously defined goals. At a higher education institution, forexample, the levels can be categorized as institutional level, programlevel, and classroom level. For example, within a program, a number ofcourses or sections may defined that are designed to satisfy theobjectives of the program, the department and the institution. Thefollowing figures in FIGS. 4A-4C depict exemplary techniques forcurriculum planning within a program.

Retuning quickly, to FIG. 3H, a user is presented a list with a numberof courses. From this list, a particular course can be selected forfurther definition. FIG. 4A depicts a user interface screen that allowsa user to define course-specific objectives for a course or coursesection within a program. Examples of a course objective, may includesuch items as critical thinking, on-line research, portrait drawing,etc. The left set of tabs 400 identify for the user that the generalfunctionality relates to curriculum planning and, more particularly, todefining section objectives.

The interface screen of FIG. 4A includes a table that lists thealready-defined objectives of this course 405. The attributes which aredefined for a course may include, but are not limited to, an objectivename 402, the location of its definition 404, the category of theobjective 406, the rubric 408, by which the objective is measured anduser access privileges for each objective. There is also a column 410that indicates how many content items have been aligned to theparticular objective of that row. Thus, when the user elects to add anobjective, using the button 403, a definition screen is presented to theuser that allows the user to enter information relating to theseattributes. For example, a form can be presented to the user with thesefields 402, 404, 406, 408, 410, 412 (or other fields as defined by theassociated template) and the user can enter data or select frompre-populated data in order to define the objective. Other attributes(not shown) may include a textual description of the objective, asummary of the objective, and key-words related to the concept of theobjective.

The “definition” attribute 404 allows a course objective to be definedin an efficient manner. For example, a particular objective may bedefined for a course (e.g., CS101) that will also be an objective forevery section of that course irrespective of the specific instructor forthe section. When the section-specific objectives are defined for asection, there is no need to re-define the particular course objectivebut rather simply select it from all the available objectives presentedon a template during the definition process. In addition, a default, orpre-selected, objective may be presented to the user as well. Byselecting an already defined objective, all the attributes can bere-used and, optionally, changed by the user if desired. Thisfunctionality also allows objectives defined for one course to beselected for inclusion in an entirely different course without repeatingthe entire objective definition process.

An objective may be aligned with any previously defined objective orgoal regardless of the operating unit in which that objective or goalwas defined (for example, see FIG. 3G and the description accompanyingthat figure). For example, a goal at one operating unit may be alignedwith an objective at an immediately lower-level operating unit which,itself, is aligned with an objective of an even lower-level operatingunit. When defining a course objective, the goals and objectives towhich it is aligned can be identified in a number of ways. A courseobjective may, for example, be aligned with a program objective whichitself is aligned with an institutional or department-level objective orgoal. In this way, there is a linking between goal an objectives fromdifferent hierarchical levels of operating units. In one example, coursecontent may be aligned with a course objective which itself is alignedwith a program objective. In this way, the course content is indirectlyaligned with program objectives (and possibly even higher-level goalsand objectives). Because goals and objectives may change over time, the“align” button 401 is provided to allow a user to update the alignmentinformation of an objective even after it has been initially defined.

The rubric attribute 408 identifies the rubric used to measure thefindings related to the objective. The present system includesfunctionality for building rubrics related to course objectives and forother uses within an assessment project. One of ordinary skill willrecognize that a variety of techniques and methods can be used toidentify the subject matter of the rubric and arrange and define itsstructure. Once created, these rubrics are managed and stored by thepresent system to be available for various uses in their respectiveoperating units. Also, assuming proper access privileges are defined,rubrics may be shared between different operating units and assessmentprojects as well. The rubrics may be general in nature such as a“writing assignment” rubric that can be applied to any writingassignments. The objectives for a particular writing assignment mayinclude, for example, “utilize research material” and “criticalthinking”. The rubric used will have a result that measures attainmentof these objectives.

Another level of assessment planning that can occur is at the lessonlevel. In other words, the lessons are identified within a course andaligned with the course or section objectives. Thus, FIG. 4B depicts aninterface screen that relates to lesson planning within a course. Withina course (e.g., Intro to Graphic design) 416, a user can define a numberof lesson plans that are meant to achieve the course objectivesidentified earlier. The table of FIG. 4B depicts a number ofalready-defined lesson plans and includes attributes of each lesson plansuch as its name 418, where it is defined 420, the number of studentresources, 422, number of faculty resources 424, the number ofassociated assessments 426, and access privileges 428. Additionalattributes may be added and some of the attributes deleted from thetable of FIG. 4B without departing from the scope of the presentinvention. Similar to methods described earlier, the user can elect toadd a new lesson plan (using the button 417) and be provided a templateappropriate for its creation. In particular, the user can be offered atemplate that allows the user to identify the attributes shown in thetable of FIG. 4B as well as the course objectives to which the lessonplan aligns. During creation of a lesson plan, the user can specify suchattributes as where, if at all, the lesson plan may already be defined,the number of course resources (e.g., books, papers, digital files,etc.) that are used within the lesson plan and the number of facultyresources used.

Within the definition of a lesson plan, there can also be theidentification of the number, names, and types of assessments that areincluded. These assessments can include tests, quizzes, papers,collaborative assignments, on-line surveys, etc.

Once the course objectives and the lesson plans have been defined, avariety of curriculum planning and curriculum mapping investigations canbe performed. Returning briefly to FIG. 4A, the column 410 identifies anumber of curriculum items or content for a course have been aligned toa particular objective of that course. Turning now to FIG. 4C, oneexample of a curriculum-related report about a operating unit isdepicted. In this report, a list of the courses 430 offered by anoperating unit (e.g., English Department) is shown. This list alsoincludes an indication 432 of the type of curriculum activities withinthe course aligned with various objectives and goals. Thus, the columnsof the list identify the activities within a course such as tests,collaboration, discussion threads, assignments, collections, etc. Foreach course in the list 430, the corresponding number of aligned itemsof each type are identified under each activity type. Thus, the reportsummarizes what types of activities in each course are aligned withvarious objectives. By selecting one of the displayed numbers 433, theuser can be presented with a detailed list of exactly which objectiveswith which that particular activity is aligned. The identifiedobjectives may be limited to merely course objectives but may alsoinclude objectives and goals from all levels (e.g., higher, lower, andparallel.)

More advanced curriculum planning and mapping may be performed as wellusing the information that the user has entered about goals, objectives,courses and lessons. For example, FIG. 4D depicts an exemplarycurriculum map 440 that graphically displays a mapping of programobjectives to course offerings. Each column 444 of the table 440 relatesto a particular program objective and each row 442 relates to a coursewithin the program. The cells of the table include an indicator 446 ofwhether or not a particular course is aligned with a particular programobjective. The indicator 446 may be a binary indicator (e.g., “yes” or“no”) or may be multi-valued. Alternative measures can be the level atwhich the objective is met (e.g., introduction, re-enforcement, mastery,etc). Thus, a user can quickly realize which objectives may not havemany courses aligned therewith or where there is a high-degree ofredundancy. As a result, curriculum planning decision can be made aboutwhat new courses might be added and what their content should be. Theobjectives just described have been identified as “program” objectivesmerely as a practical example. However, one of ordinary skill willrecognize that these objectives may also be goals and objectives fromhigher, lower, and parallel levels in the institution hierarchy.

Another possible curriculum mapping function can include a differentview of courses and aligned objectives. For example, an operating unitthat “owns” a course (e.g., the Computer Science Department) can use theinformation described above to identify all other units that havealigned program objectives to that course. Thus, the curriculum plan canbe viewed from both a top-down approach as well as a bottom-up approach.Potential for new courses or changes to existing courses within aprogram can be determined by identifying which goals and objectives arebeing ignored by the offerings within a program. Furthermore, within thecourses themselves, lesson plans can be modified or augmented to moreclosely align with the course objectives (which indirectly align to theother levels of the hierarchy). From the top level of the institution,evidence can be automatically gathered from across the various operatingunits of how the institution's goals and objectives are being met. Thisevidence is collected automatically, systematically, and objectivelyduring the performance of the assessment projects and is readilyavailable for internal evaluation by the institution as well as byexternal accreditation boards.

Instrument Creation and Management

FIG. 5A depicts a flowchart of an exemplary method for deployinginstruments within the assessment system 102 described herein inaccordance with the principles of the present invention. In general, aninstrument is a data collection tool that can include, for example,surveys, quizzes, portfolios, assignments, assignment binders, papers,tests. These instruments are used to collect data that is used forvarious purposes within an assessment project. Some data may haverubrics applied to it, other data may be used to measure achievement ofa goal or objective, and other data may be used to quantify values beingassessed within the project. For purposes of explaining FIG. 5A, thespecific example of a test as the instrument will be used without anintent to limit instruments to only tests.

In step 502, a user identifies a new test instrument through the userinterface. This is typically performed by making a selection to create anew test or to edit an existing test. The user continues, in step 504,by defining the contents of the test, or instrument, itself. Because ofthe hierarchical arrangement of some of the operating units within thepresent system, it is advantageous that a test can incorporate defaultinformation required by a department or the institution. For example,every test may require the student to sign a “honor” clause or to answercertain demographic information. Thus, when a new test is defined, itwill automatically include such information. In the example where theinstrument is a survey, the institution may have certain requiredquestions and each department may have its own auxiliary questions.Thus, the starting “canvas” for a survey instrument may initially have avariety of default questions depending on the unit under which theinstrument is created.

In step 506, the recipient list for the instrument is selected. Thislist may be generated based on data about user, units or courses. Thelist might be generated dynamically at the time of deployment or havebeen generated previously and saved as a static list. For example, therecipient list may be defined as all 400-level sections taught by atenured professor. If generated dynamically, that list would show all400-level sections taught by a tenured professor at the time ofdeployment, therefore the dynamically generated list might be differentwhen used the Fall of 2006 and the Fall of 2007, as the sections taughtby tenured faculty will have changed. However, if generated staticallyin the Fall of 2006, it will store all 400-level sections taught by atenured professor at the time of the list creation. That list would bethe same in Fall of 2007 as it was in Fall of 2006, because itrepresents a static view of the data as it was at the time oflist-generation. Furthermore, an instrument may be copied and thenmodified such that the modified copy is deployed to an earlierpopulation or a different population altogether. Accordingly, both theoriginal instrument and the changed copy are available for deployment inthe future.

Deployment of an instrument can occur in a variety of ways, some ofwhich rely on the academic system 116 of FIG. 1, in particular, thecommunity system portion of the academic system 116. As mentionedbriefly, and described in more detail in the incorporated patentapplication, the academic system and community system include a virtualspace that each student may visit to send and receive information anddeployment of an instrument is often accomplished within thisenvironment. In one particular implementation, the academic system mayinclude logical components such as a learning system, a content system,and a transaction system such as those offered commercially by thepresent Assignee. Typically, there is a home page that a user ispresented after logging in to the academic system 116. From the homepage, the user can navigate to a “My Courses” area, an “E-Mail” area, a“Financial” area, etc. and from the “My Courses” area, the user can openrespective windows specific to each course in which the student isenrolled. Thus, deployment of an instrument can entail sending anexecutable web module that displays on the student's home page or withinthe appropriate course area. The deployment could also be accomplishedthrough a simple HTML link, or an e-mail message. The deployment (forexample, in the case of employer surveys of recent graduates) may alsoinclude sending information to parties external to both the assessmentsystem 102 and the academic system 116. Advantageously, the instrumentcan be deployed in more than a single media (e.g., off-line via bubblesheets) as well depending on the recipients preferences or some othersetting within the assessment system 102.

FIG. 5B depicts a screenshot of a user interface for defining testinstruments for an English department. The exemplary attributes of theinstrument that are explicitly shown include the instrument name 510,the number of available versions 512, information about previousdeployments 514, and access privileges 516. Other attributes may includethe deployed date, the due date, and a textual description of theinstrument. Each instrument has a selection link that allows the user toselect the instrument for further action (e.g., remove) and a “Manage”button 520 for editing and management. Additionally, there is a “deploy”button for when the user wants to actually deploy a previously definedinstrument. If multiple versions of an instrument are available, the“deploy” button may cause an intermediary window to inquire of the useras to which version to deploy. As part of the deployment, the user ispresented with an interface screen (not shown) that allows selection ofthe desired recipients.

Adding a new test instrument is initiated by selecting the “Add” button519. Doing so causes a new test creation page to be presented to theuser for modification and editing. From this from a user can define suchitems as the questions of the test, how it is scored, the instructionsfor completing the test, and similar test characteristics.

FIG. 5C is similar to FIG. 5B except that the items in column 530 referto survey instruments rather than test instruments. FIG. 5D depicts anexemplary form defining a survey instrument that might be offered to auser when that user elects to define a new survey. For example, the formidentifies the survey name 532 and includes a brief summary area 534about the survey itself. The particular example of FIG. 5D includes atechnique 536 for breaking the survey into sections in which differentquestions 538 can be entered. As shown, the questions can be relativelycomplex with multiple scoring criteria. The depicted survey form isexemplary in nature and one of ordinary skill will recognize that manyother types and forms of questions may be used without departing fromthe scope of the present invention.

The present assessment system 102 also provides, via its user interface,methods for managing course evaluation instruments for a particularoperating unit. For example, the interface screen 540 of FIG. 5Eprovides a list 542 of course evaluations available to the EnglishDepartment. The displayed list of available evaluations can be modifiedby selecting the appropriate level of detail from the window 544. Thus,a user may select (if access privileges allow) evaluations from otherunits and even from higher-level operating units. From within themanagement screen 540, the user can select options to manage 550 theevaluation or to deploy 548 the evaluation. Because multiple deploymentsof the same evaluation are possible, the screen 540 includes deploymentstatistics 546 about this particular evaluation.

The user can elect to create a new course evaluation by selecting the“Add” button 552. Similar to techniques already described, anappropriate authoring form is provided to the user in order to definethe new evaluation. An exemplary authoring form is depicted in theinterface screen 560 of FIG. 5F. The authorship of an evaluation can bedistributed among various authors because some questions may be from theinstitution, others may be from the department, still others may be fromthe program, some may be course specific and still others may beinstructor specific. As explained earlier, the selection of recipientsof an instrument (e.g., an evaluation) can be based on user demographicinformation, course attribute information, or a combination of both.Additionally, course evaluations may be delivered such that a recipientis presented with an aggregation of questions that is determineddynamically by the recipient's context (e.g., institution, college,department, course, section, sex, ethnicity, residence status, etc.).

Because the assessment system 102 already includes a variety ofinformation and data that is typically used to perform questionnairefunctions within the institution, a questionnaire manager may beprovided that defines standard questionnaires and their contents. Thus,when a user desires to generate a questionnaire, it can be initiallycreated within the assessment system 102 using the templates maintainedby the questionnaire manager. The benefits of such a system include theadvantageous result that a department, for example, will receivesimilarly formatted reports and content from each of its separateprograms. The exemplary interface screen of FIG. 5G depicts aquestionnaire generator that provides a selection window 562 thatpermits a user to define new fields of the questionnaire. When a newfield is added, then its label, instructions and other attributes can bedefined as well. The screen view of FIG. 5G shows a list 564 of all thedefined fields for the current questionnaire. A preview mode (not shown)is available to show how the questionnaire layout looks for presentationto a user.

One of ordinary skill will recognize that there are similarities betweena survey and other instruments for collecting data. Unlike a survey,some data collection instruments are not delivered to a specific set ofusers and may be open for anyone to complete. For example, asatisfaction survey would be sent to a specific set of users typicallyall at the same time and users would only be able to submit once.However, a curriculum change request form would be available for a userto complete at any time, as determined by the user's need to request achange in the curriculum (e.g., a faculty requesting a new course beadded to the curriculum), and the user may submit it multiple times asthe need to provide the data may arises on multiple occasions. The datacollection capabilities of the survey and course evaluation tools can beused for such general purpose data collection instruments as well, witha change to the deployment model. Instead of being deployed to aspecific set of users, such a general purpose data collection instrumentcould be made accessible to a set of users for submission at thediscretion of the user based on the user's needs. The exemplaryinterface screen of FIG. 5G depicts a generator of such a generalpurpose data collection instrument that provides a selection window 562that permits a user to define new fields of the instrument. When a newfield is added, then its label, instructions and other attributes can bedefined as well. The screen view of FIG. 5G shows a list 564 of all thedefined fields for the current instrument. A preview mode (not shown) isavailable to show how the layout looks for presentation to a user.

The availability of data across a wide variety of units, courses, andusers raises concerns of uniform consistent and efficient evaluation ofthe data. Thus, an evaluation portfolio manager is provided for anoperating unit that lists evaluation templates available to users withinthe operating unit. Typically, within a higher-education institution,for example, the evaluation template manager would be at a relativelyhigh level such as at the college or department level. An evaluationportfolio relates to a particular portion of an assessment project. Forexample, a college may have to satisfy three different accreditationboards to be “accredited”. A different portfolio could be created foreach of the different accreditation boards. Each portfolio would haveidentified, therein, a series of objectives (as outlined by theaccreditation board). Drilling down into each objective would be anidentification of how “satisfaction” of each objective is measured (asurvey, a rubric for writing samples, and a test). The next step then isto collect the artifacts that pertain to each of these measures. Thus, aportfolio includes a label, a number of objectives, one or more ways tomeasure instruments related to each objective, and the artifactsthemselves. At least three different types of assessment capabilitiesare provided in conjunction with evaluation portfolios. For example, astudent or faculty may perform self-assessment. Additionally, a facultymember, for example, may use an evaluation portfolio to performassessment of an individual (e.g., a student) with respect to a courseobjective, one or more program objectives, an institutional goal etc.Furthermore, the evaluation portfolios may aggregate results from manyentities within the institution to provide assessment of institutionalor program-level objectives such as, for example, accreditationcriteria.

The interface screen of FIG. 5H depicts an exemplary evaluationportfolio manager that lists the available evaluation portfolios. Inthis example, one evaluation portfolio 566 is listed and is related toNCATE accreditation. The screen further displays attributes of thisportfolio such as where it is defined 568, and its deployments 570. Fromthis screen, the user can manage 573 a portfolio, add 571 a newportfolio, or deploy 570 an existing portfolio.

Selecting the “Add” button 571 of the portfolio manager interface willreveal the interface screen of FIG. 5I that shows a template of aportfolio that includes a listing of the areas that are the subject ofthe evaluation portfolio. This template can then be modified by theuser. A portfolio is intended to include artifacts from students thatexhibit certain outcomes. For example, artifacts may be papers, tests,surveys, quizzes, collaborative exercise, assignments, etc. Theseartifacts are submitted by a student (or collected by the faculty) inorder to be applied towards one or more of the areas identified withinthe evaluation portfolio template. In the example portfolio 566, thereare four areas of interest for evaluation. A list of these areas 580 andtheir respective descriptions 582 are shown. By selecting one of thelinks 580, a more detailed view of the specific properties for thatportfolio area can be provided to the user.

This evaluation portfolio is deployed by a user to selected recipientsusing the “deploy” button 572 shown in FIG. 5H. As described earlier, alist of recipients may be defined or selected for deployment that variesaccording a number of criteria. Once the evaluation portfolios aredeployed to students within the academic system 116 (see FIG. 1), thesestudents can submit artifacts related to the different areas. Thus, thestudents to which the evaluation portfolio is deployed each complete theevaluation portfolio. Completing the portfolio is accomplished by addingartifacts to the portfolio. As a result, the assessment system 102collects artifacts from a number of students related to the evaluationareas 580 of the evaluation portfolio. Because, the artifacts caninclude identifying and other meta-data, robust filtering and searchingof the submitted and stored artifacts are possible. For example, randomsamples of submitted artifacts may be selected for an area or,alternatively, only artifacts from female students that live off-campusmay be selected.

When defining the evaluation portfolio, the user may also be given theopportunity to define the artifacts that should be collected for each ofthe areas. Thus, by selecting one of the areas 580, the user may beprovided with an artifact definition template that specifies attributesof the artifact (e.g., the scoring system, the fields, and the date).One example area in a portfolio may be to show that a student nurse canperform an intramuscular delivery of medicine. The artifact fordocumenting this area may require the name of the managing clinicalnurse, a statement that the student was observed performing theinjection, the name of the hospital, and the date of the observation.Another area in a portfolio may relate to satisfying a creative writingrequirement. The artifact to apply to this area may include a paper thatwas scored according to a particular rubric. Thus, the artifactsthemselves may vary greatly depending on the area 580 within anevaluation portfolio. Accordingly, the present assessment system 102allows a user to define templates for the different types of artifactsand then, subsequently, when creating an evaluation portfolio, allowsthe selection of these templates to identify an artifact when describingan area within the evaluation portfolio, such as that in FIG. 5I.

A concept that is somewhat similar to a portfolio is that of anassignment binder. An assignment binder typically pertains to aparticular accreditation program and collects artifacts related to thatpurpose over the course of one or more terms. The contents of theassignment binder can be used to support a review of a particularprogram or curriculum. However, unlike a portfolio, an assignment bindersimply collects the artifacts without necessarily assigning them toparticular areas or measuring them against predetermined rubrics. FIG.5J depicts an exemplary interface screen for managing assignmentbinders. As shown, a binder is created within the College of Engineeringand includes a name 590 and a description 592. Other attributes of abinder include, for example, the term for which it applies and useraccess privileges. Through the use of user lists and course lists, theselection of the artifacts to include in the binder can be controlled toprovide desired cross-sections of available artifacts. Although anassignment binder may include artifacts from a wide range of students,selective filtering based on user lists, unit lists, or course lists canbe used to extract a limited subset of artifacts from students matchingparticular criteria. Additionally, artifacts can be selectively returnedthat match a filter, or search criteria, related to the grade of anassignment, or other assignment specific attributes. Thus, differentbinders may be built, from the same pool of artifacts, by automaticallyretrieving artifacts matching one or more defined filter criteria.Within the interface of FIG. 5J, more information and the artifactsthemselves may be presented to a user through selection of one of thelinks 590.

The definition of goals and objectives, the creation of rubrics, and thecollection of artifacts as described above allow automatic evaluation ofa student's performance against program or course objectives. Theassessment system 102 and its integration with the other campus systems114 and the campus academic system 116 simplify sharing of informationamong the various systems. As a result, the integration of these systemsresults in the storage of rubrics, artifacts, various objectives,different goals, and a definition of the relationship or associationbetween these different data entities. Using techniques known to one ofordinary skill, this integrated information can be organized andfiltered to identify and report various aspects related to the storeddata that may be pertinent to assessment within an institution.

For example, a “score-sheet” such as that depicted in FIG. 6A may bebuilt as the result of the integration and flexibility provided by thepresent assessment system. This table of FIG. 6A includes a list ofstudents 602 and an indication of how they satisfy each objective 604,606. The indication 608 may be a numerical score or some otherqualitative-type score. For example, in the table of FIG. 6A, theindicators include “below”, “meets”, and “exceeds”. These indicators arebased on the definition of the rubrics and the scores that are assignedto the artifacts which the students supplied in response to the rubrics.Because the rubrics are aligned with previously defined goals andobjectives, these indicators can be automatically determined from theseobjectives. The table of FIG. 6A also includes a drop down window 610that allows a user to select other objectives to display (e.g., criticalthinking) as columns in the table.

FIG. 6B depicts a flowchart of an exemplary method by which the table ofFIG. 6A may be generated. In step 620, instruments, rubrics, and goalsand objectives are defined as described earlier. As part of thedefinition process, the instruments are associated with differentrubrics and these rubrics are aligned with various goals and objectives.Next, the instruments (e.g., test and quizzes) are deployed, in step622, to students. Other activities that are aligned with various rubricsmay be performed as well.

In response to the deployment of instruments and other activities,results are received from the students, in step 624, by the assessmentsystem. The receipt of results may be through automatic submission bythe students through a campus academic system 116 as described earlieror through an instructor who receives the results directly from thestudents. The rubrics may then be applied to the results, in step 626.Once the rubric is applied, scores can be assigned to the results foreach of the defined goals and objectives so that a report can begenerated (see FIG. 6A), in step 628. The results and assigned scoresalong with other data may then be stored, in step 630, so that recordsof the assessment may be maintained for future reference and analysis.

Reporting

The following figures depict a variety of different reports that may begenerated using the assessment tools described herein and the data thatis collected as a result of using these assessment tools. One ofordinary skill will recognize that the specific format and contents ofeach report may vary greatly without departing from the scope of thepresent invention. However, these reports show the great breadth anddepth of the types of data collection, some of which may be automatic,and analysis provided by the assessment techniques and methods describedherein. One advantageous benefit of the present system and techniquesevidenced within the exemplary reports is the sharing of data from amongdifferent level operating units. For example, the data from a number ofdifferent course may be “rolled-up” into a report that is generated at adepartment level and this report may include data that is collected atthe course or section level. Thus, a report related to a department goalor objective may include results from artifacts that were submitted atthe course level even though there are intervening operating unitsbetween the two levels within the model of the institution. This sharingof data between different operating units and different levels ofoperating units occurs in the other direction as well. For example, whendefining and identifying artifacts, instruments, surveys, portfolios,curriculum content, etc. at a lower-level operating unit attributes fromhigher-level operating units may be automatically inherited without theneed to be explicitly defined

FIG. 7A depicts a list 702 of report templates that have beenpre-defined for use by a user. As new reports are defined, they can beadded to the list. In operation, a user would select one of theavailable templates and then fill out the fields of the template withthe particular data for their report. To help in the selection of thereport, the list can also include other identifying features such as adescription 704 associated with each report template.

One example report is shown in FIG. 7B (which is a more detailed viewFIG. 4C); this report displays a table of “Course vs. Objective”. Ingenerating this table, the English Department, for example, may use thetemplate to select some or all of its courses and then select all orsome of its department objectives. Because the definition of the coursesalso included aligning it with certain objectives, this report can begenerated to show how each course satisfies each objective. The tableincludes an indicator 706 at each intersection of an objective andcourse. This indicator could be quantitative such as a ranked numericalscore or it can be qualitative as shown in FIG. 7B. Exemplaryqualitative categories are shown in the legend 708 of the table.

FIG. 7C depicts a table that might be useful to the institution whichshows, by department, the status of course evaluations. Because thegeneration and return of course evaluations are automatically andsystematically handled by the present assessment system, thesestatistics are readily available without manual collection and tallyingof results by each department. Furthermore, not only is the status ofcompletion revealed in the table but the overall scores 710 of theevaluations can be calculated and displayed as well.

In addition to tabular reports, more graphical report templates areprovided as well. FIGS. 7D and 7E depict a dashboard-type report thatgraphically indicates certain indicators that an institution or otheroperating unit may be interested in. In a dashboard, certain indicatorsare defined along with the attributes for that indicator. For example,the “traffic light” indicator 716 of FIG. 7D has three possible states(e.g., read, yellow, green) which are mapped to a particular graduationrate defined by the institution. The data collection described hereinprovides the raw data for this indicator and the definition of theindicator is used to map the raw data into the “color” of the trafficlight. Similarly, a pie chart 718 can be created from raw data to showthe percentage of students who passed and failed a course. Similar rawdata may be displayed as a bar chart 720 as well. The templatedefinition screen of FIG. 7E depicts one way in which such a dashboardmay be created. For example, a user can create a new metric 714 usingthe “Add” button 713. In the creation process, the user is provided witha selection window 712 that selects the type of indicator associatedwith the metric.

FIG. 7F depicts another type of report that resembles a scorecard. Thistype of report may be advantageously used to compare results of the samemetric over different instances. For example, the scorecard of FIG. 7Fshows such statistics as student SAT scores, student satisfaction, thenumber of minority faculty, etc. However, from an institutionperspective, the scorecard provides the values for these metrics for thefour most recent terms 720. Thus comparison and tracking of differentmetrics may be accomplished using a scorecard. FIG. 7G shows the detailsof using a scorecard template to generate a scorecard such as the one inFIG. 7F.

Using this template, a user can define the metric 722 that is going tobe applied and displayed along with the class of individuals 724 to whomit applies. A description 726 is useful for identifying the metric and aselection window 728 allows the user to define the type of data thatresults from applying the metric. For example, the data may be“dollars”, “a percentage”, “raw numbers”, “an average”, etc. Near thebottom of the template of FIG. 7G is a window 730 that allows the userto select the different time frames for which these metrics will beincluded in the scorecard.

It is useful, once a report has been defined and run, to be able to savethe report for future reference. As shown in FIG. 7H, the assessmentsystem 102 described herein provides that capability. When storing thereports, it may be advantageous to allow the user to set permissions 732for the report such that certain classes of users may be prevented fromaccessing the report or, alternatively, to specify a wide group of userroles that may access the report so as to ensure its widestdissemination.

FIG. 7I depicts a relatively simple bar chart; however, its contentsreveal the great breadth of data collection and reporting capabilityprovided by the assessment system 102 described herein. At theinstitution, one level of operating units may be “the college”. Thedifferent colleges 736 may include the business college, the engineeringcollege, etc. Within each college are courses and each of these courseshave course evaluations that are deployed and collected and given arating 738. Each course also has a particular number of seats orstudents and this data may be stored in a third-party campus informationsystem maintained by the institution or it may be stored within thepresent assessment system. As a result, the bar graphs of FIG. 7I allowa reporting of data such that the institution can discover the averagecourse evaluation ratings within each different college as a function ofclass size. The legend 734 depicts an exemplary breakdown of classsizes.

FIG. 7J depicts another type of graph 741 that can be used to displaydata. In this instance the data is from an off-campus survey thatcollected ratings 740 from employers of recent graduates. Additionally,the graphs are broken down by different programs 742 within the collegeof engineering. Thus, as an operating unit, the college of engineeringcan analyze data according to sub-units defined previously.

For purposes of auditing and other monitoring functions, the assessmentsystem 102 can track activities of users within the system in order toknow what documents they have accessed and modified. Thus, the table ofFIG. 7K can be created that shows recent events 744 within theassessment system 102 that are associated with one or more users 746.This data may be filtered so as to create a report that is based ontime, based on a particular user, based on a class of users, etc.

Other Tools

There are a number of additional administrative type functions thatsimplify the management of various assessment projects and initiatives.These functions are not required by the assessment system 102 describedherein but facilitate its operation and provide advantages to automatingthe processes described herein.

For example, users can be grouped into committees in order to simplifyassigning roles and access privileges. Templates and other documents,when created, may have access privileges assigned based on whichcommittee may access them, approve them, modify them, etc. Byassociating staff and faculty into different defined committees, theaccess privileges remain consistent even though faculty and staff canchange. Additionally, upon creation, a committee can be provided withits own discussion board area, its own shared calendar, its own e-mailcontact list, as well as its own storage repository for the documents itneeds and generates. The interface screen of FIG. 8A depicts a tooluseful for listing existing committees 804 and adding 802 newcommittees.

FIG. 8B depicts a task manager function which allows an operating unitto identify tasks 804 and track their status. FIG. 8C depicts some ofthe detailed information that may be collected about a student orfaculty. This information may be manually entered or populated fromanother system such as an SIS of the institution. Information such asgender, ethnicity, financial aid, native language, SAT scores, etc. maybe collected. As a result, the user lists described earlier can befinely defined to include only a desired subset of students whendeploying instruments or reporting results and other data. Some of thefields shown in FIG. 8C may be static while others are dynamic. Thus, asdata changes (e.g., the students GPA), this data may be automaticallypulled from the SIS when needed. The assessment system 102 can accessthe SIS system in at least two different ways. A snapshot of the SIS canbe made that the assessment system 102 stores and uses for its purposesor the SIS system can be dynamically accessed each time data from it isneeded.

Because of the automatic and systematic collections of data describedherein, there will be a large number of files generated and stored.Organization of these files can be dependent on operating units,assessment projects, programs, courses, instrument type, etc. FIG. 8Ddepicts an exemplary file manager that may be used to utilize the fileswithin the system for the English Department 808. Within the managerwindow, a list of file collections 810 can be provided along withassociated attributes 811. To help locate file collections, a drop downwindow 812 may be provided to allow the user to apply certain filters sothat only desired file collections are displayed.

FIGS. 8E-8H depict a template manager that allows operating units todefine the templates that will be used in assessment projects and tolist existing templates 814. Using this manager, a user can modify 815an existing template or add 816 a new template. One attribute of atemplate is the owner 818 of the template which reflects the operatingunit that created the template. As shown in FIG. 8E, an owner may be anoperating unit hierarchically above the present operating unit.Accordingly, the user of the “college of Liberal Arts”, for example, canstart with a template provided by “the Institution” and addcollege-specific revisions. In this way, the institution can havecomplete or flexible control over the contents of assessment projecttemplates to ensure some level of consistency across the differentsub-operating units but also allow customization.

FIG. 8F depicts a interface screen that allows creation of an assessmentproject template. Using this interface screen, the user can specify whatforms 820 are to be included in a template and view a description 822 ofeach form. For example, returning briefly to FIG. 3G, the names in thecolumn 820 correspond to the form names that are along the top tabs252-262. As shown, however, the exact form names may vary as well as thenumber of forms defined within a template. Exemplary forms shown in FIG.8F include:

Objectives: The objectives for the assessment project

Measurements: The measurements that will be used to measure theobjective (e.g., test, survey, portfolio)

Action Plan: The ordered steps to be taken within the Assessment Project

Findings: The findings resulting from the measurements

Follow-up: The steps for a follow-up plan, if any.

The user is presented with a way to define these forms to handle eachdata collection process. Thus, when a user initiates the start of a newassessment project, the templates that are provided to the user arethose templates and forms that are defined in this process. One ofordinary skill will recognize that other or additional forms may be usedas well to define an assessment project template without departing fromthe scope of the present invention.

FIG. 8G depicts an exemplary form definition interface for the“Objectives” form. A similar form definition interface is also providedfor the other forms identified as well. The interface screen of FIG. 8Gallows a user to add fields 824 to the form which define the way theform is presented when displayed for a user to use when defining a newassessment project. The interface screen of FIG. 8G shows how thedetails of the form's fields are defined and specified. For example,field 7 is provided with a user configurable “type” 826, and “label”828. Additionally, the user defines (for this particular field), the“drop-down items” 830 from which the user can select. The specificfields and their attributes are not critical to an understanding of thepresent assessment system 102 as one of ordinary skill will recognizethere are many functionally equivalent ways to allow users to organizeand define templates and the forms and fields that make up a template.However, the results are advantageously depicted in FIG. 3G where theparticular fields of FIG. 8G result in the template screen that was seenearlier in FIG. 3G.

Within an institution, there may be many different operating units, someof which have different terms. For example, the medical school, the lawschool, and the engineering department may all have different termlengths and days. Thus, instruments or portfolios that must be completedby the end of a term or the middle of a term may have different duedates for different operating units. A term manager is thereforeprovided that allows a user to define different terms. In the interfacescreen of FIG. 8H, the user is presented with a way to specify a termname 832 and assign to it a start date 834 and an end date 836.

One advantageous function described previously related to user lists andtheir use when deploying instruments and portfolios. In particular, auser list is a multi-faceted concept that refers to both its definitionand its members. Thus, a user list may be defined as “all male juniorsin the College of Engineering” and the members which satisfy thatdefinition are also referred to as the “user list”. Referring back toFIG. 8C, a variety of user demographic information is collected andmaintained by the assessment system. This information can be employed todefine very specific subsets of students by presenting the fields ofFIG. 8C to a user as selectable filters which the user can configure todefine the desired members for a particular user list.

Once the user list is defined, the user has the option of making it astatic list or a dynamic list. If it is a static list, then the user canapply the definition at a particular date to select the members of theuser list. When some other process references this user list, then onlythose members within the original list will apply (e.g., only thosemembers will receive a particular course evaluation). If it is a dynamiclist, then the selected members are determined each time the user listis referenced. In other words, a survey deployed to a particular dynamicuser list may be sent to different members if sent on two differentdates.

The user manager interface screen of FIG. 8I provides a way for a userto maintain, define, and select different user lists. Using an “Add”button 844, a user is presented a new definition screen which allowsthem to configure the filters for selecting the desired subset of users.Once defined, this user list can be named and stored so that it isavailable in the list of FIG. 8I. This list provides an identification840 of available user lists along with a number 842 of the memberswithin the user list.

Similar to a user list, the assessment system 102 also provides the userwith a way to selectively define course lists. A course list includesthose courses that match a specified set of criteria. The attributes ofa course (to which the criteria are applied) may be specified and storedwithin the assessment system 102 or be imported from another campuscomputer system such as an online catalog or similar database. Thebreadth of different course lists that can be created is limited only bythe possible number of permutations allowed by the different attributesused to describe a course. Thus, very finely defined course lists may becreated to allow selective deployment of portfolios, course evaluations,and other surveys. The course list manager interface screen of FIG. 8Jdepicts an exemplary way for a user to be presented with existing courselists and to define a new course list. Although not shown, creating anew course list involves a user being presented with an interface thatallows defining the attributes and their values which must be satisfiedto be a member of that particular course list. Once defined, then adynamic or static course list can be created and stored.

Because operating units may be hierarchically arranged, it is possibleto take advantage of that organization to simplify the deployment ofcertain instruments/survey and to perform certain reporting. Forexample, in an institution made of multiple campuses, each with multiplecolleges, it is possible to ensure the same survey is deployed to allcolleges of a campus simply by specifying the campuses without the needto specify each individual college. This functionality is provided byallowing the user to define unit lists. A unit list is simply thoseoperating units that meet particular criteria. The interface screen ofFIG. 8K depicts a unit list manager tool that displays defined unitlists 860. Selecting one of those lists 860 will display the names ofthe operating units within that unit list. Using the screen of FIG. 8K,the user may also elect to define a new unit list as well.

By automating the process of defining and collecting information relatedto assessment projects, a vast data warehouse can be generated thatprovides many different views of the assessment process. Searchingthrough that data warehouse using selectable criteria allows a user tocreate a view of the data that may be useful for a variety of differentpurposes. One of ordinary skill will recognize that there are numerousways to categorize and store data to aid in searching and locatingpertinent information. The search interface screen of FIG. 8L is merelyone exemplary way in which a view of the data can be defined andretrieved from within the assessment system 102 described herein. Usingthis screen, the user can select from a multitude of search criteria862. This search criteria can include, for example, unit names 864,keywords, 866, author 868, dates 869, objectives 870, aligned goals 872,etc.

Assessment System and Academic System

The previously-described functionality of the assessment system 102 hasbeen focused on the assessment system in isolation from many of theother systems available within the institution or organization. However,as briefly mentioned with respect to FIG. 1, a number of other systemsmay exist within the institution that may advantageously be integratedwith the present assessment system. As mentioned earlier, the assessmentsystem 102 described herein can operate in conjunction with an academicsystem as described in the previously incorporated patent and patentapplications. This Internet-based learning system is also called theacademic system 116 (See FIG. 1) in that it provides a virtual, on-linecommunity for students. FIG. 9A depicts an exemplary screen that a usermay encounter when logged into the academic system 116.

In FIG. 9A the user is provided with electronic tools 902 such ase-mail, calendar, and a directory. The academic system 116 also allowsthe user to connect to external campus computer systems using links 904such as the Registrar's office or the Advising office. Of particular useto a student, the “My Courses” tab 906 allows the user to drill downinto the courses for which they are enrolled to see course assignments,lecture notes, external resources, discussion boards etc. Additionally,system wide announcements can be provided in a window 908.

Specific to the assessment system functionality, two modules 910 and 912are displayed on the user's interface page 900. The module 910 describesan assessment exercise that a student must complete by a certain date.It includes a link 911 that the user can select to be taken to theassessment exercise. The implementation of the exercise may be on someother computer system that receives and records the user's responses.These responses and scores are automatically provided to the assessmentsystem 102 and associated with that particular user for that particularassessment exercise. The other module 912 relates to one or more surveysthat the user is to complete. Similar to the assessment exercise, thestudent selects and then completes the survey with the resultsautomatically returned to the assessment system.

The modules 910, 912 refer to instruments that are defined by a user ofthe assessment system 102 using the techniques previously described. Theuser of the assessment system 102 also employs the user list or courselist (or both) to select the group of students to receive the respectiveinstruments and then deploys the instruments. Once deployed, theassessment system 102 sends notification to the academic system tolocate those students selected to receive the instrument. Within theacademic system, a notation is made within the records associated withthe implicated students regarding the newly available instrument. Whenthe students next log in to the system, their respective home pages arerendered and the included notation results in a display of theappropriate modules 910, 912. An alternative deployment method would beto add a new instrument within the course area of a particular course.Thus when a student drills down to that course, the course page will berendered and it will display a link to any new instruments.

Faculty members also use the academic system 116 and may be interestedin assessment projects within their department or involving the studentpopulation that they interact with. For example, window 920 of FIG. 9Bprovides, for the faculty member, links to information on unit forms,surveys, course evaluations, results, tests, etc. Window 922 relates toa particular assessment project (e.g., “2006 Undergraduate DegreeReview”) and provides links to information such as the objectives andfindings collected so far with respect to this project. Access to thisinformation is provided through appropriate communications interfacesbetween the assessment system 102 and the academic system. Furthermore,user roles and identities can be used to ensure information is presentedto only those users having the required access privileges.

As mentioned, within the course-specific areas that a student may accessfrom their virtual space within the academic system, a student hasaccess to a variety of information pertaining to that course such asassignments, reading lists, discussion boards, lecture notes, etc. Withthe connection to the assessment system 102 in place, the student maynow also be provided with a listing of the goals and objectives alignedto each course. Additionally, by drilling down into each lesson withinthe course, the student can see what are the objectives of today's orsome other day's lesson. Additionally, objectives and goals can bealigned to individual content items within the course. For example, aparticular discussion board topic may be presented to satisfy aparticular course or department objective. This alignment can bespecified by the instructor and displayed within the course area visibleto the student.

The academic system 116 includes both a content management system and anadministrative system. The assessment system 102 described previouslycan advantageously utilize this functionality already provided by theacademic system. For example, the assessment-related content may beorganized by operating unit and managed and accessed using the sharedcontent window of the academic system. FIG. 9C shows the shared contentwindow 929 that includes a unit content tab 930. The unit content tab930 can include hierarchically arranged content within the unit asshown. Selection of specific content from the content tab 930 willdisplay the associated file collections within a sub-window 932. Fromthere, the file hierarchy can be traversed. The display of content inFIG. 9C may be logical in nature and does not necessarily reflect theactual hierarchical arrangement of files as stored within the underlyingfile system. The window 933 of FIG. 9D depicts an administrative screenof the academic system 116. Rather than having a separate administrativefunction for the assessment system, it can be provided through a section934 of the window 933. As shown, an administrator can manage functionsrelated to assessment projects, curriculum planning, lesson planning,testing, surveys, reporting etc. While not necessary, the integration ofthe assessment system 102 with the academic system 116 simplifiessharing of user roles, content, course attributes and also simplifiesdeploying instruments and collecting responses. All these activities canstill be accomplished with diverse, separate systems but areadvantageously simplified the tighter the two systems are integrated.

Template Driven

In the above-described assessment system 102, the use of templates hasbeen highlighted as an advantageous technique for defining andcollecting the data to be used in assessment projects. For example, atthe institutional level a template is provided which guides the userinto providing strategic goals and mission and vision statements. Whenan assessment project is begun, the user is presented with templatesthat allow objectives to be identified, measurements to be defined, andaction plans to be specified. When the identified objectives are to bedefined, a template is provided to guide the user through this processand to align them with higher level goals and objectives. At the unitlevel, templates are provided to define portfolios, instruments,programs and program objectives. While at the course level, otherappropriate templates are presented to a user to help develop lessonplans, instruments and course and content objectives. In each instance,the lower level objectives and outcomes can be aligned with higher-levelobjectives and goals. As a result, a template driven system is providedthat defines objectives at least at three different hierarchical levelsand allows automatic alignment of the objectives throughout the threelevels. Of course, more levels are possible as well, because the presentassessment system provides operating units with the freedom to modeltheir organization an a flexible manner.

Workflow

The different phases of an assessment project can be considered asevents within the assessment system 102. As such, a workflow system maybe included to monitor and facilitate an assessment project. Forexample, certain re-occurring assessment projects may be tied to thestart or end of a term. Thus, a calendar function within a workflowprocess can initiate a creation of an appropriate assessment projecttemplate and forwarding it to the initial user responsible for managingthe project. For such a project, as well as those manually initiated,the subsequent steps of the assessment project may be handled by aworkflow process. The task plan described earlier can include with eachtask the identity of the responsible parties and the disposition of theproject after completion of each task. For example, once an exemplaryassessment project reaches the stage where all of a department'sobjectives have been defined, then a subsequent step, for example, maybe to create a curriculum map for that department. Through use of aworkflow process this automatic stepping through the task plan may beaccomplished. Some workflow steps may merely include sending e-mails tocommittee members regarding the status of some phase of an assessmentproject. Another benefit of implementing a workflow system is that partsof the assessment project may have its access privileges automaticallymodified at each step of the task plan. For example, once a set ofprogram objectives are agreed upon by the committee, then theseobjectives become read-only and no one has permission to modify themThus, because of the automatic and systematic collection ofassessment-related information and tasks available within the assessmentsystem 102 described herein, an automatic workflow capability may beincluded to further automate and monitor an institution's assessmentinitiatives.

Export and Import

Assessment projects, and portions thereof, are data entities in and ofthemselves. For example, an English Department can create an assessmentproject that relates to how its courses align with its objectives. Thecreation of such a project includes a definition of objectives andgoals, an organization of operating units, a definition of courses, andan identification of course objectives. Inherent in this data is how allthe objectives from the different hierarchical levels align with oneanother. If further detail is desired, each course can includeinformation on course contents and instruments deployed within eachcourse. Another assessment project may be related to how a departmentprepared for a particular accreditation review. This might includeevaluation portfolios, surveys, and other components. In each of thesecases, other institutions may benefit from copying those assessmentplans. Thus, portions of an assessment project (or the entire project)may be copied within the institution for use by various programs anddepartments while such functionality may also occur across differentinstitutions as well.

Thus, the presently described assessment system 102 can export anassessment project or only portions of an assessment project as a dataentity that can be transferred to another institution having its ownassessment system. One example of a portion of an assessment projectthat may be exported, or imported, may be a series of templates relatedto National Council for Accreditation of Teacher Education (NCATE)accreditation. At that second institution, then, the assessment systemcan import the data entity and have a pre-populated assessment plan touse or to modify. At a smaller level, a publisher of a course or programof courses can define the objectives met by its courses and theirrespective content as well as the instruments and, possibly, portfoliosthat might accompany such a program of courses. This information may beimported into the assessment system at a department or program level atthe institution to avoid developing the same material on their own fromscratch.

Within such imported or exported data there may be references tothird-party standards. If a user imports a project that includes areference to, for example, an objective of the standard (e.g.,analytical reasoning), that user will most likely desire to know exactlywhat that standard says about that objective. Thus, importing andexporting data within the assessment system 102 provides the additionalfunctionality to include external standards. This functionality may alsoextend to exporting and importing instruments related to measuring orevaluating these standards as well.

One result of having such import and export functionality includedwithin the present assessment system is that repositories of assessmentproject-related data entities may be created. Thus, assessment projects,templates, instruments and other data may be exported to and stored in arepository that is accessible by other users via a network or othercomputer system. Such a repository of information may include differentlevels of anonymity. For example, an institution may want to export tothe repository information about the incoming freshman class' SATscores, high-school GPA, and student age that was collected during aparticular assessment project. This information can then be used byother institutions for comparison and analysis. In such an example, thefirst institution may want to have their identity associated with thedata but also insure that the individual student information isanonymous. In a second example, the institution itself may want toremain anonymous in interacting with the repository. In this case, theinstitution may provide one or more aspects of an assessment project tothe repository which can then provide a comparison against a benchmarkor other standard. In this way, the repository provides the institutionwith feedback regarding the assessment project (e.g., whether or not thestudents performance on a battery of tests were below, at, or above anationwide average). Alternatively, the institution can access therepository and import different benchmarks and other standards toperform their own comparison and analysis. Because virtually any aspectof an assessment project may be exported to the repository, institutionscan exchange and compare various instruments, rubrics, surveys,templates, reports, as well as results collected in performing theassessment project. Furthermore, this repository may also includenon-assessment related information and data as well such as, forexample, course content and individual academic portfolios.

Multi-Institution

Not only can the assessment plans and portions of the assessment plans(e.g., the objectives, the courses) be exported and shared, the resultsand findings can be shared as well. Accordingly, a multi-collegeconsortium, for example, can request and receive findings from itsmembers and then report on the findings from the overall perspective. Ifthe members of the consortium utilize substantially similar assessmentprojects and initiatives, then the collected data and findings will beconsistent in nature and format and will allow them to be simplyamalgamated and analyzed.

A number of variations to the specific behaviors and steps described inthe above examples may be made without departing from the scope of thepresent invention. The various illustrative logical blocks, modules,circuits, elements, and/or components described in connection with theembodiments disclosed herein may be implemented or performed with ageneral purpose processor, an application specific integrated circuit(ASIC), a field programmable gate array (FPGA) or other programmablelogic component, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. A general-purpose processor may be a microprocessor,but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine.

The methods or algorithms described in connection with the embodimentsdisclosed herein may be embodied directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in RAM memory, flash memory, ROM memory,EPROM memory, EEPROM memory, registers, hard disk, a removable disk, aCD-ROM, or any other form of storage medium known in the art. A storagemedium may be coupled to the processor such that the processor can readinformation from, and write information to, the storage medium. In thealternative, the storage medium may be integral to the processor.

The previous description is provided to enable any person skilled in theart to practice the various embodiments described herein. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other embodiments. Thus, the claims are not intended to belimited to the embodiments shown herein, but is to be accorded the fullscope consistent with the language claims, wherein reference to anelement in the singular is not intended to mean “one and only one”unless specifically so stated, but rather “one or more.” All structuraland functional equivalents to the elements of the various embodimentsdescribed throughout this disclosure that are known or later come to beknown to those of ordinary skill in the art are expressly incorporatedherein by reference and are intended to be encompassed by the claims.Moreover, nothing disclosed herein is intended to be dedicated to thepublic regardless of whether such disclosure is explicitly recited inthe claims. No claim element is to be construed under the provisions of35 U.S.C. §112, sixth paragraph, unless the element is expressly recitedusing the phrase “means for” or, in the case of a method claim, theelement is recited using the phrase “step for.”

1. A method of modeling a plurality of organizational levels in support of conducting an assessment within a multi-level institution, the method comprising the steps of: receiving first input from a user selecting a node associated with one of the plurality of organizational levels; presenting the user with an editable template for the node; and modifying the node by completing at least a portion of the editable template based, at least in part, on second input received from the user.
 2. The method of claim 1, further comprising the steps of: for the node, creating for and presenting to another user at least one customizable assessment project associated with the node; and completing the at least one customizable assessment project based, at least in part, on other input received from the another user.
 3. The method of claim 2, wherein the user and the another user are the same.
 4. The method of claim 1, further comprising the step of: storing the modified node along with any previously defined nodes to create a model of the multi-level institution.
 5. The method of claim 1, wherein the multi-level institution comprises at least three organizational levels including multiple operating units.
 6. The method of claim 5, wherein the at least three organizational levels further comprise a programs level, a course level, and a section level.
 7. The method of claim 1, further comprising the step of: selecting the editable template from a plurality of editable templates.
 8. The method of claim 7, wherein the editable template is selected based on one or more criteria, said criteria including: the one of the plurality of organizational levels associated with the node; a type of operating unit associated with the node; an identity of the node; a type of operating unit hierarchically above the node; whether the node is a program, a course, or section, and an identity of an operating unit hierarchically above the node.
 9. The method of claim 7, wherein respective contents of each of the plurality of editable templates are based on at least one of: its associated organization level; a type of operating unit associated with the node; an identity of the node; a type of operating unit hierarchically above the node; an identity of an operating unit hierarchically above the node; and whether the node is a program, a course, or section.
 10. The method of claim 1, wherein each of the organization levels have associated therewith, respectively, at least one node.
 11. A system for modeling a plurality of organizational levels in support of conducting an assessment within a multi-level institution, the system comprising: a computer-based storage including a plurality of editable templates; and a programmable computer, having access to the computer-based storage, and configured to: receive first input from a user selecting a node associated with one of the plurality of organizational levels; present the user with one of the plurality of editable templates for the node; and modify the node by completing at least a portion of the editable template based, at least in part, on second input received from the user.
 12. The system of claim 11, wherein the programmable computer is further configured to: create and present to another user at least one customizable assessment project associated with the node; and complete the at least one customizable assessment project based, at least in part, on other input received from the another user.
 13. The system of claim 12, wherein the user and the another user are the same.
 14. The system of claim 11, wherein the programmable computer is further configured to: store the modified node along with any previously defined nodes to create a model of the multi-level institution.
 15. The system of claim 11, wherein the multi-level institution comprises at least three organizational levels including multiple operating units.
 16. The system of claim 15, wherein the at least three organizational levels further comprise a programs level, a course level, and a section level.
 17. The system of claim 11, wherein the programmable computer is further configured to: select the one editable template from the plurality of editable templates.
 18. The system of claim 17, wherein the programmable computer is further configured to select the one editable template based on one or more criteria, said criteria including: the one of the plurality of organizational levels associated with the node; a type of operating unit associated with the node; an identity of the node; a type of operating unit hierarchically above the node; whether the node is a program, a course, or section, and an identity of an operating unit hierarchically above the node.
 19. The system of claim 17, wherein respective contents of each of the plurality of editable template are based on at least one of: its associated organization level; a type of operating unit associated with the node; an identity of the node; a type of operating unit hierarchically above the node; an identity of an operating unit hierarchically above the node; and whether the node is a program, a course, or section.
 20. The system of claim 11, wherein each of the organization levels have associated therewith, respectively, at least one node.
 21. A method for creating customizable templates for modeling organizational levels related to an assessment within a multi-level institution, the method comprising the steps of: receiving first input from a user identifying one of the organizational levels; receiving second input from the user identifying one or more attributes associated with each of a plurality of nodes associated with the one organizational level; and creating an editable template, based on the second input, to be used to model one of the plurality of nodes of the assessment.
 22. The method of claim 21, wherein the one or more attributes are based on the associated one organizational level.
 23. The method of claim 21, wherein the one or more attributes are based on one of: the one organizational level; a type of operating unit associated with the one of the plurality of nodes; an identity of the one of the plurality of nodes; a type of operating unit hierarchically above the one of the plurality of nodes; an identity of an operating unit hierarchically above the one of the plurality of nodes; and whether the one of the plurality of nodes is a program, course, or section.
 24. The method of claim 21, further comprising the steps of: presenting the editable template to another user; and defining the one of the plurality of nodes by completing at least a portion of the editable template based, at least in part, on additional input from the another user.
 25. The method of claim 24, wherein the user and the another user are the same.
 26. The method of claim 21, further comprising the steps of: presenting the editable template to another user; and modifying the one of the plurality of nodes by completing at least a portion of the editable template based, at least in part, on additional input from the another user.
 27. The method of claim 26, wherein the user and the another user are the same.
 28. The method of claim 21, further comprising the step of: storing the editable template for retrieval.
 29. A system for creating customizable templates for modeling organizational levels related to an assessment within a multi-level institution, the system comprising: a programmable computer configured to receive first input from a user identifying one of the organizational levels; receive second input from the user identifying one or more attributes associated with each of a plurality of nodes associated with the one organizational level; and create an editable template, based on the second input, to be used to model one of the plurality of nodes when performing an assessment project.
 30. A method of modeling a plurality of organizational levels in support of conducting an assessment within a multi-level institution, the method comprising the steps of: receiving first input from a user selecting a first node associated with one of the plurality of organizational levels; presenting the user with an editable template for a second node; and defining the second node by completing at least a portion of the editable template based, at least in part, on second input received from the user.
 31. The method of claim 30, wherein the first node is hierarchically above the second node within the plurality of organizational levels.
 32. The method of claim 30, wherein the first node is hierarchically immediately above the second node within the plurality of organizational levels
 33. The method of claim 30, further comprising the step of: storing the second node along with any previously defined nodes to create a model of the multi-level institution.
 34. The method of claim 30, wherein the multi-level institution comprises at least three organizational levels including multiple operating units.
 35. The method of claim 34, wherein the at least three organizational levels further comprise a programs level, a course level, and a section level.
 36. The method of claim 30, further comprising the step of: selecting the editable template from a plurality of editable templates.
 37. The method of claim 36, wherein the editable template is selected based on one or more criteria, said criteria including: a respective one of the plurality of organizational levels associated with the first or second node; a type of operating unit associated with the first or second node; an identity of the first or second node; a type of operating unit hierarchically above the first or second node; whether the first or second node is a program, a course, or section; and an identity of an operating unit hierarchically above the first or second node.
 38. The method of claim 36, wherein respective contents of each of the plurality of editable templates are based on at least one of: its associated organization level; a type of operating unit associated with the first or second node; an identity of the first or second node; a type of operating unit hierarchically above the first or second node; an identity of an operating unit hierarchically above the first or second node; and whether the second node is a program, a course, or section.
 39. A system for modeling a plurality of organizational levels in support of conducting an assessment within a multi-level institution, the system comprising: a computer-based storage including a plurality of editable templates; and a programmable computer, having access to the computer-based storage, and configured to: receive first input from a user selecting a first node associated with one of the plurality of organizational levels; present the user with one of the plurality of editable templates for a second node; and create the second node by completing at least a portion of the editable template based, at least in part, on second input received from the user.
 40. The system of claim 39, wherein the programmable computer is further configured to: store the second node along with any previously defined nodes to create a model of the multi-level institution.
 41. The system of claim 39, wherein the multi-level institution comprises at least three organizational levels including multiple operating units.
 42. The system of claim 41, wherein the at least three organizational levels further comprise a programs level, a course level, and a section level.
 43. The system of claim 39, wherein the programmable computer is further configured to: select the one editable template from the plurality of editable templates.
 44. The system of claim 43, wherein the programmable computer is further configured to select the one editable template based on one or more criteria, said criteria including: a respective one of the plurality of organizational levels associated with the first or second node; a type of operating unit associated with the first or second node; an identity of the first or second node; a type of operating unit hierarchically above the first or second node; whether the first or second node is a program, a course, or section; and an identity of an operating unit hierarchically above the first or second node.
 45. The system of claim 43, wherein respective contents of each of the plurality of editable template are based on at least one of: its associated organization level; a type of operating unit associated with the first or second node; an identity of the first or second node; a type of operating unit hierarchically above the first or second node; an identity of an operating unit hierarchically above the first or second node; and whether the second node is a program, a course, or section.
 46. The system of claim 39, wherein the first node is hierarchically above the second node within the plurality of organizational levels.
 47. The system of claim 39, wherein the first node is hierarchically immediately above the second node within the plurality of organizational levels.
 48. Computer readable media containing programming instructions for modeling a plurality of organizational levels in support of conducting an assessment within a multi-level institution, that upon execution thereof, causes one or more processors to perform the steps of: receiving first input from a user selecting a node associated with one of the plurality of organizational levels; presenting the user with an editable template for the node; and modifying the node by completing at least a portion of the editable template based, at least in part, on second input received from the user.
 49. Computer readable media containing programming instructions for modeling a plurality of organizational levels in support of conducting an assessment within a multi-level institution, that upon execution thereof, causes one or more processors to perform the steps of: receiving first input from a user selecting a first node associated with one of the plurality of organizational levels; presenting the user with an editable template for a second node; and defining the second node by completing at least a portion of the editable template based, at least in part, on second input received from the user.
 50. Computer readable media containing programming instructions for creating customizable templates for modeling organizational levels related to an assessment within a multi-level institution, that upon execution thereof, causes one or more processors to perform the steps of: receiving first input from a user identifying one of the organizational levels; receiving second input from the user identifying one or more attributes associated with each of a plurality of nodes associated with the one organizational level; and creating an editable template, based on the second input, to be used to model one of the plurality of nodes of the assessment. 